Is the deconvolution layer the same as a convolutional layer?

1 In our CVPR 2016 paper [1], we proposed a novel network architecture to perform single image super­resolution (SR). Most existing convolutional neural network (CNN) based super­resolution methods [10,11] first upsample the image using a bicubic interpolation, then apply a convolutional network. We will refer to these types of networks as high­resolution (HR) networks because the images are upsampled first. Instead, we feed the low­resolution (LR) input directly to a sub­pixel CNN as shown in Fig.1 : Figure 1: An illustration of the ESCPN framework where r denotes the upscaling ratio. Let denote the upscaling ratio ­ e.g if the input LR image is then the output HR image will be r 1 × 1. We then output number of channels instead of one high­resolution (HR) image and use periodic r × r r 2 shuffling to recreate the HR image. The exact details about how our efficient sub­pixel convolutional layer works can be found in the paper. We will refer to our network as a LR network. In this note, we want to focus on two aspects related to two questions most people asked us at CVPR when they saw this network. Firstly, how can channels magically become a HR image? And secondly, r 2 why are convolution in LR space a better choice? These are actually the key questions we tried to answer in the paper, but we were not able to go into as much depth and clarity as we would've liked given the page limit. To better answer these questions, we first discuss the relationships between the deconvolution layer in the form of the transposed convolution layer, the sub­pixel convolutional layer and our efficient sub­pixel convolutional layer, which we'll go through in Sec. 1 and Sec. 2. We will refer to our efficient sub­pixel convolutional layer as a convolutional layer in LR space to distinguish it from the common sub­pixel convolutional layer [5]. We will then show that for a fixed computational budget and complexity, a network with convolutions exclusively in LR space has more representation power at the same speed than a network that first upsamples the input in HR space.